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SUMMARY 

Psychophysiological  monitoring  of  operator  state  has  been  developed  over  the  past  decades  as  measures  of 
mental  workload,  fatigue,  and  inattention.  This  paper  provides  examples  of  the  application  of  these  measures 
to  classify  operator  functional  state  (OFS)  and  further  to  implement  adaptive  aiding.  Examples  using  several 
levels  of  mental  workload  are  presented  to  show  that  combinatorial  classifiers  can  utilize  the  information 
from  several  psychophysiological  measures  to  provide  highly  accurate  correct  classification  of  OFS. 
An  example  of  adaptive  aiding,  using  on-line  assessment  of  OFS,  demonstrated  that  highly  accurate 
classification  could  be  achieved  and  that  using  this  information  to  control  adaptive  aiding  enhanced 
performance  of  a  complex  task.  A  model  of  how  psychophysiological  and  performance  data  could  be  used  to 
provided  continuous  OFS  monitoring  and  adaptive  aiding  is  presented. 


INTRODUCTION 

The  goal  of  automation  is  to  improve  overall  system  performanee.  There  are  several  methods  used  or 
proposed  to  implement  automation  (Billings,  1997;  Parasurman,  2000  &  Rouse,  1991).  One  method  of 
automation  implementation  is  aeeomplished  by  making  it  an  integral  part  of  the  system  so  that  the  automation 
is  eontinuously  applied.  Other  methods  of  automation  implementation  involve  the  operator  to  differing 
degrees.  Adaptive  aiding  in  particular  applies  the  automation  based  upon  various  contingencies.  This  may 
include  an  assessment  of  the  operator’s  functional  state.  For  example,  if  the  operator  is  fatigued  or  inattentive 
the  automation  may  engage  the  operator  in  an  additional  task  in  order  to  overcome  this  negative  state  that  may 
lead  to  errors.  In  situations  where  the  demands  of  the  task  have,  or  are  about  to,  overwhelm  the  cognitive 
capabilities  of  the  operator  the  system  may  assume  control  of  certain  tasks  in  order  to  reduce  the  demands  on 
the  operator.  If  successful,  the  operator  will  then  be  able  to  manage  the  remaining  tasks  and  avoid  errors  that 
would  jeopardize  the  mission.  Successful  implementation  of  adaptive  aiding  depends  on  (1)  highly  accurate 
knowledge  of  the  current  functional  state  of  the  operator  and  (2)  implementation  procedures  that  assist  and  do 
not  interfere  with  the  operators  job  performance. 

The  first  requirement  is  the  highly  accurate  assessment  of  the  momentary  functional  state  of  the  operator. 
To  be  accepted  by  users,  operator  functional  state  (OFS)  assessment  must  be  highly  accurate  and  reliable. 
If  the  OFS  assessment  is  not  accurate  then  aiding  will  be  implemented  at  inappropriate  times  and  will  not  be 
provided  when  needed.  Operators  will  not  use  adaptive  aiding  schemes  that  do  not  improve  their  performance 
and  insure  higher  overall  mission  success.  Optimal  functional  state  refers  the  ability  of  the  operator  to 
successfully  meet  the  demands  of  the  job.  Operator  state  is  seen  as  a  broader  condition  such  as  awake,  asleep. 
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and  fatigued.  Operator  functional  state  refers  to  whether  or  not  the  operator  ean  meet  the  demands  of  the  job 
at  that  moment.  Being  awake  does  not  mean  that  an  operator  is  eurrently  eapable  of  optimal  performance. 
For  example,  an  operator  may  be  awake  but  inattentive  and  therefore  has  a  high  probability  of  missing 
important  signals  which  can  result  in  errors.  That  is,  the  operator’s  functional  state  relative  to  the  demands  of 
the  job  is  not  appropriate.  Because  job  demands  vary  a  great  deal,  the  relationship  between  the  job  and  the 
operator’s  state  is  dependent  upon  how  well  the  operator  is  able  to  function  at  any  given  moment  in  the 
context  of  the  requirements  of  the  job  (see  figure  1).  In  this  context  it  is  then  very  important  that  the  operator’s 
functional  state  is  accurately  assessed  with  regard  to  the  immediate  demands  of  the  job.  Most  aiding  systems 
infer  the  state  of  the  operator  from  a  model  of  a  “typical”  human  operator.  This  model  is  provided  with  the 
current  mission  conditions  and  the  appropriate,  ideal  operator  behavior  is  estimated.  If  the  actual  operator  is 
not  performing  according  to  the  “task  goals”  or  preplanned  expectancies  then  aiding  is  implemented. 
Automation  is  then  implemented  based  upon  the  inferred  needs  of  the  operator  in  the  current  situation. 
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Figure  1:  Schematic  of  Adaptive  Aiding  System  with  Psychophysioiogicai  and  Performance  Inputs 
Using  Context  information  from  Mission  Requirements.  OFS  is  used  to  determine  if  aiding  is  required. 


In  order  to  provide  a  more  accurate  estimate  of  the  status  of  OFS  we  use  psychophysioiogicai  measures  that 
are  collected  and  evaluated  in  real-time  (Wilson,  Lambert  &  Russell,  2000).  This  psychophysioiogicai 
assessment  provides  additional  input  to  the  adaptive  aiding  system.  This  makes  it  possible  to  include  the 
actual  operator  status  in  the  equation  and  should  enhance  the  performance  of  the  state  estimator.  When  the 
psychophysioiogicai  signals  indicate  that  the  operator  is  in  an  unfavorable  state,  aiding  can  be  implemented  in 
an  attempt  to  bring  the  operator  to  an  optimal  state.  Psychophysioiogicai  measures  can  be  used  to  detect 
states  of  fatigue  and  inattention  on  one  hand  and  cognitive  overload  on  the  other.  The  psychophysioiogicai 
signals  are  continuously  present  making  uninterrupted  on-line  assessment  possible.  Incorporating  the 
psychophysioiogicai  assessment  with  the  performance  based  assessment  can  lead  to  more  accurate  estimates 
of  the  operator’s  functional  state  (Wilson,  &  Russell,  1999). 
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PSYCHOPHYSIOLOGICAL  ASSESSMENT  OF  OFS 

A  large  body  of  research  results  confirms  the  utility  of  psychophysiological  measures  as  metrics  of  OFS 
(Kramer,  1991;  Wilson  &  Eggemeier,  1991;  Wilson,  2002a;  Wilson,  2002b).  A  large  proportion  of  these 
investigations  are  from  aviation  research  and  are  derived  from  laboratory,  simulator  and  flight  settings 
(Wilson,  2001).  Cardiac,  eye,  brain  and  muscle  activity  can  easily  be  monitored  while  operators  perform  their 
jobs.  Taken  separately  these  measures  provide  valuable  data  about  the  impact  of  job  performance  on  the 
operator.  They  are  especially  valuable  when  taken  in  the  context  of  actual  job  performance.  Because  most  jobs 
involve  complex,  multiple  task  performance.  Single  task,  laboratory  based  research  is  important  in  deriving 
basic  concepts.  Flowever,  multiple  task  performance  may  produce  different  relationships  among  the  variables 
than  is  found  in  the  simpler,  single  task  research.  This  lays  the  foundation  for  deriving  OFS  assessment  which 
combines  several  psychophysiological  measures  in  the  context  of  complex  task  performance.  We  have  been 
successful  using  this  paradigm  and  have  demonstrated  high  levels  of  accuracy  in  correlating  changes  in 
psychophysiological  measures  with  task  demands  using  methods  which  include  several  psychophysiological 
measures.  Typically  the  correct  classification  accuracies  of  OFS  range  from  85%  to  98%  (Russell,  Wilson  & 
Monet,  1996;  Russell  &  Wilson,  1998;  Wilson  &  Fisher,  1991;  Wilson  &  Russell,  1999;  Wilson,  Lambert  & 
Russell,  2000). 


CLASSIFICATION  ACCURACY 

An  example  of  this  approach  is  from  the  Russell  and  Wilson  study  (Russell  &  Wilson,  1998).  U.  S.  Air  Force 
air  traffic  controllers  were  trained  to  stable  performance  on  a  simulated  air  traffic  control  task.  Task  difficulty 
was  manipulated  along  two  dimensions,  the  volume  of  aircraft  and  the  overall  complexity  of  the  situation. 
Each  of  the  two  dimensions  provided  three  levels  of  task  demand.  The  levels  were  confirmed  by  the 
controllers  performance  scores  and  their  subjective  ratings  of  the  tasks  difficulty.  A  third  manipulation  was 
designed  to  provide  very  high  levels  of  mental  workload.  Physiological,  performance  and  subjective  data 
were  collected.  Data  from  seven  air  traffic  controllers  were  used  to  train  and  test  two  classifiers. 
Stepwise  discriminant  analysis  and  artificial  neural  network  (ANN)  classifiers  were  used  to  determine  how 
well  they  could  correctly  classify  the  mental  workload  of  the  air  traffic  controllers.  The  classifiers  were 
trained  using  a  portion  of  the  physiological.  The  trained  algorithms  should  be  able  to  recognize  the  response 
patterns  associated  with  low  and  high  task  demand  conditions.  Then,  the  remaining  data  were  provided  to  the 
classifiers  which  were  required  to  determine  if  these  data  were  from  the  high  or  low  task  demand  conditions. 
Correct  classification  accuracies  ranging  84%  to  88%  across  the  seven  subjects.  A  further  step  employed 
feature  reduction  using  saliency  analysis  techniques  for  the  ANNs  resulted  in  a  mean  of  90%  correct 
classification  accuracy.  Because  the  most  critical  discrimination  in  actual  operational  situations  would  be 
between  the  overload  condition,  which  would  result  in  possibly  catastrophic  errors,  and  the  other,  manageable, 
workload  conditions  a  two  class  problem  was  tested.  The  three  levels  of  volume  or  complexity  task  difficulty 
were  collapsed  into  one  condition  and  compared  with  the  overload  condition  data.  That  is,  the  low,  medium 
and  high  data  were  combined  and  equal  exemplars  from  each  made  up  the  “normal”  data  set.  The  ANN  was 
trained  with  subsets  of  the  “normaf  ’  and  overload  data.  The  remaining  data  were  used  to  test  the  accuracy  of 
the  trained  classifier.  This  resulted  in  almost  perfect  classification  accuracies  with  mean  percent  correct  of 
98%.  These  results  are  very  encouraging  because  in  real-world  situations  the  most  important  distinction 
between  operator  functional  states  would  be  to  detect  mental  overload  states.  These  results  demonstrate  that 
psychophysiological  data  can  be  used  to  accurately  assess  the  cognitive  demands  placed  upon  operators. 
Further,  the  psychophysiological  data  are  continuously  available  which  permits  moment-to-moment 
assessment  of  OFS.  Currently  available  physiological  sensors  are  easily  applied  and  accepted  by  operators. 
Numerous  signal  processing  algorithms  are  available  to  detect  and  correct  artifacts  in  the  physiological  data. 
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Current  state-of-the-art  hardware  and  software  technology  provide  fast,  light  weight  components  which  permit 
real-time  collection,  analysis  and  classification  of  operator  functional  state.  This  coupled  with  the  availability 
of  inexpensive  memory  and  disk  storage  provides  adequate  computing  power  for  on-line  assessment  of  OFS 
assessment. 

Accurate  OFS  assessment  is  the  first  step  in  achieving  a  viable  adaptive  aiding  system.  The  classification  of 
OFS  must  be  highly  accurate  in  order  to  provide  useful  information  and  also  to  gain  user  acceptance.  If  the 
assessment  of  OFS  is  erroneous  much  of  the  time  then  it  provides  little  benefit  to  the  user.  In  order  to  achieve 
wide  user  acceptance  correct  OFS  assessment  will  probably  need  to  be  in  the  high  ninety  percent  correct 
range.  Otherwise  it  will  add  little  to  making  the  decision  to  implement  aiding  and  may  cause  errors 
when  inappropriately  implemented.  Lower  levels  of  correct  assessment  may  be  acceptable  if  the 
psychophysiological  assessment  is  but  one  of  a  number  of  assessment  methods  being  used  to  decide  when 
adaptive  aiding  should  be  implemented.  In  a  voting  system,  with  a  number  of  input  variables, 
the  psychophysiological  assessment  could  be  used  as  one  component  to  derive  a  composite  OFS  score. 


ADAPTIVE  AIDING 

The  second  requirement  for  a  practical  adaptive  aiding  system  is  the  appropriate  means  of  implementing  the 
aiding.  A  growing  body  of  automation  literature  includes  discussion  of  ways  in  which  automation  should  be 
implemented  (Billings,  1997;  Parasuraman,  2000;  Rouse,  1991;  Scerbo,  1996).  The  possible  methods  seem  to 
range  from  employing  the  automation  at  all  times  to  implementing  the  aiding  only  when  necessary  based  upon 
situational  and/or  operator  variables.  The  methods  based  upon  situational  and/or  operator  variables  can  result 
in  the  implementation  of  aiding  as  soon  as  it  is  deemed  necessary,  when  an  agreed  upon  situation  is  met  which 
has  been  prearranged  with  the  operator.  Alternatively,  deleterious  OFS  could  be  signaled  to  the  operator  as  a 
warning  so  that  the  operator  could  decide  on  actions  to  take  to  improve  the  situation. 

Laboratory  tasks  have  been  used  to  demonstrate  that  psychophysiologically  determined  adaptive  aiding 
improves  operator  performance  (Pope,  Bogart,  &  Bartolome,  1995;  Prinzel,  Freeman,  Scerbo,  Mikulka  & 
Pope,  2000;  Wilson,  Lambert  &  Russell,  2000).  The  Pope  and  Prinzel  studies  used  LEG  to  monitor  subject’s 
engagement  in  a  task.  When  the  subjects  state  were  deemed  to  not  be  properly  engaged  in  the  task  a  second 
task  was  added  to  improve  their  attention  to  the  tasks.  Wilson,  Lambert  &  Russell,  (2000)  used  brain,  eye, 
heart  and  respiration  measures  to  determine  when  subjects  were  performing  the  more  difficult  task  level  of  a 
complex  task.  The  Multi-Attribute  Task  Battery  (MATB)  was  used  to  provide  the  complex  task.  The  MATB 
consists  of  four  separate  subtasks.  The  subtasks  are:  monitoring,  manual  tracking,  resource  management  and  a 
communication  task.  In  the  monitoring  task,  when  one  of  two  lights  change  color  the  subject  is  required  to 
position  a  cursor,  using  their  right  hand,  over  the  light  and  click  a  mouse.  There  were  four  dials  which  were 
monitored,  when  any  of  the  dial  pointers  moved  out  of  range,  the  subject  clicked  the  mouse  in  that  dial. 
A  manual  tracking  task  was  controlled  via  a  joystick  controlled  by  the  right  hand.  The  resource  management 
task  required  the  subject  to  maintain  the  fluid  level  of  two  tanks  by  turning  pumps  on  and  off  to  supply  these 
two  main  tanks.  In  the  communication  task  the  subject  responded  to  their  call  sign  by  changing  the  frequency 
setting  of  one  of  four  channels.  Reaction  times  and  error  scores  were  recorded  as  the  subjects  performed  the 
task.  Seven  subjects  practiced  the  task  until  they  reached  stable  performance.  Three  levels  of  task  difficulty 
were  presented  to  the  subjects.  The  lowest  level  required  the  subjects  to  only  look  at  the  MATB  screen. 
The  second  level  required  the  subjects  to  perform  the  task  at  a  low  level  of  difficulty.  Events  occurred  fairly 
infrequently,  about  1 0  seconds  apart.  In  the  third  level  the  density  of  event  occurrence  was  much  higher  and 
could  not  be  performed  without  making  errors.  The  subjective  reports  substantiated  that  the  levels  were 
significantly  different.  The  subjects  performed  the  group  of  three  difficulty  levels  three  times.  During  the  first. 
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the  psychophysiological  data  were  eollected  and  subsequently  provided  to  an  ANN  to  train  it  to  diseriminate 
among  the  three  levels.  The  ANN  was  trained  separately  for  eaeh  subject.  On  the  second  time  the  tasks  were 
presented  the  ANN  classified  the  task  levels  using  the  psychophysiological  data  based  upon  the  recent 
training.  During  the  final  performance  of  the  group  of  tasks,  the  ANN  classified  the  tasks,  on-line.  If  the  high 
task  difficulty  level  was  detected  adaptive  aiding  was  implemented  by  removing  two  of  the  four  subtasks. 
The  results  of  the  second,  classification  only,  session  resulted  in  overall  mean  correct  classification  of  the 
tasks  as  follows.  The  baseline  level  was  correctly  classified  85%  of  the  time,  the  low  task  difficulty  82%  of 
the  time  and  the  high  difficulty  level  at  86%.  These  are  significantly  above  the  33%  expected  by  chance  alone. 
The  implementation  of  the  adaptive  aiding  resulted  in  44%  reduction  of  the  tracking  error  and  33%  reduction 
of  the  resource  management  error.  Taken  together  these  results  demonstrate  that  very  high  levels  of  accuracy 
can  be  accomplished  using  complex  tasks  and  that  implementation  of  adaptive  aiding  based  upon  this 
classification  does  not  interfere  with  task  performance.  In  fact,  task  performance  significantly  improved  with 
the  aiding.  While  this  is  only  one  example  of  on-line  OFS  classification  with  associated  adaptive  aiding  it 
does  suggest  that  this  will  be  a  fruitful  line  of  endeavor. 


MODEL  OF  AN  ADAPTIVE  AIDING  SYSTEM 

A  model  of  how  OFS  assessment  could  be  used  to  initiate  adaptive  aiding  is  shown  in  Figure  one. 
The  operator’s  psychophysiology  and  their  performance  would  be  continuously  monitored.  Their  situational 
awareness  (SA)  would  be  evaluated  in  light  of  the  current  mission  requirements.  The  psychophysiological  and 
SA  evaluations  would  be  used  to  determine  their  functional  state.  The  current  mission  requirements  would 
provide  the  context  with  which  to  determine  if  the  current  OFS  was  appropriate  or  if  adaptive  aiding  was 
required  to  achieve  optimal  operator  and  system  performance.  For  example,  different  psychophysiological 
responses  and  SA  would  be  expected  during  the  cruise  portion  of  a  flight  compared  to  the  responses  expected 
during  aircraft  landing.  If  aiding  is  required,  it  would  be  applied  in  a  manner  previously  agreed  upon  with  the 
operator.  The  form  and  timing  of  the  aiding  would  also  be  consistent  with  the  immediate  mission 
requirements.  For  example,  in  some  situations  advisory  warnings  would  be  sufficient  if  the  current  mission 
requirements  were  not  time  critical.  In  other  situations,  aiding  would  be  immediately  implemented  because 
delay  would  result  in  costly  or  catastrophic  errors.  If  the  psychophysiological  and  SA  responses  were 
inconsistent  with  the  current  mission  segment  then  a  decision  to  implement  adaptive  aiding  would  be  made. 
This  could  result  in  the  system  assuming  certain  functions  in  order  to  relieve  the  workload  on  the  operator. 
If  the  aiding  successfully  changed  the  OFS  to  a  more  optimal  level  then  the  functions  assumed  by  the  system 
would  later  be  returned  to  the  operator.  The  monitoring  of  OFS  would  be  done  continuously  throughout  the 
mission.  The  consequences  of  high  and  low  cognitive  workload,  fatigue  and  attention  would  be  continuously 
monitored  with  adaptive  aiding  implementation  when  necessary  to  keep  the  operator  in  an  optimal  state. 
This  would  enhance  mission  success  by  keeping  the  key  element,  the  human  operator,  at  their  peak 
performance  throughout  the  mission. 

Application  of  this  technology  to  actual  systems  will  require  consideration  of  the  needs  of  the  operator  and 
mission.  Operator  acceptance  of  this  technology  will  be  based  upon  demonstrated  validity  and  reliability  of 
the  OFS  assessment  methods.  Further,  the  method  of  implementation  of  adaptive  aiding  must  be  thoughtfully 
considered.  The  aiding  must  be  applied  in  a  fashion  that  does  not  interfere  with  the  operator’s  performance. 
The  aiding  cannot  hinder  the  operator’s  access  to  information,  i.e.,  automatically  overlaying  the  system 
monitoring  screen  with  unnecessary  menus.  The  type  of  aiding  should  be  applied  with  the  prior  consent  of  the 
operator.  These  are  examples  of  the  many  factors  that  must  be  considered  when  the  implementation  of 
adaptive  aiding  is  considered. 
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